Project Proposal: Group 5¶
Group members: Rashi Selarka, Ryan Liu, Prem Phillips, Steve Wang
library(plyr)
library(tidyverse)
library(repr)
library(tidymodels)
library(GGally)
library(dplyr)
library(digest)
library(testthat)
library(readxl)
library(RColorBrewer)
Introduction¶
Knowledge retention continues to be a topic of great interest in pedagogical research. There are multiple clauses of knowledge retention that researchers have studied in the past, such as what kind of teaching methods would lead to greater retention (Beers & Bowden, 2005), or a study by Mueller and Oppenheimer (2014) examined whether taking notes on a laptop or longhanded led to better retention of knowledge. In this project, we're using data collected on students' study times as well as their exam scores on the topic of "electric DC machines" to gauge their knowledge levels. The dataset has a categorical variable called the knowledge level of the user (UNS), which is a measurement of how well a student understands the course. We will attempt to create a classification model to predict knowledge levels.
How can we predict a student's knowledge level from the degree they: studied for goal object materials, repeated goal object materials, and performed on exams for goal objects?
This data was donated to UCI Machine Learning by Hamdi Tolga Kahraman (2009) from their PhD thesis.
Methods: Preliminary Analysis¶
The dataset is freely available on UCI ML's learning repository and is an excel file. It is easy to upload into R and, in fact, has already been largely tidied and split into training and testing sets. There are 5 predictor variables we will be examining: study time for goal material, amount of repetitions for it, study time for related material, exam performance for goal material, and exam performance for related material.
Note: It isn't clarified what the related object materials are, but we assume since the goal object is DC machines, we can assume related objects would include other concepts from electrophysics such as AC machines.
We will narrow down 2-3 relevant predictor variables upon further analysis detailed below. We will begin by loading the data into R:
url <- "https://archive.ics.uci.edu/ml/machine-learning-databases/00257/Data_User_Modeling_Dataset_Hamdi%20Tolga%20KAHRAMAN.xls"
download.file(url, destfile = "data_set.xls")
knowledge_data_training <- read_xls("data_set.xls", sheet = "Training_Data") %>% select(STG, SCG, STR, LPR, PEG, UNS) %>% mutate(UNS = as_factor(UNS))
knowledge_data_testing <- read_xls("data_set.xls", sheet = "Test_Data") %>% select(STG, SCG, STR, LPR, PEG, UNS)%>% mutate(UNS = as_factor(UNS))
levels(knowledge_data_testing$UNS) <- c("very_low", "Low", "High","Middle") #Testing set names did not match training set names
knowledge_data_testing$UNS <- factor(knowledge_data_testing$UNS, levels = c("very_low", "High","Low", "Middle"))
head(knowledge_data_training)
print('Table 1')
head(knowledge_data_testing)
print('Table 2')
nrow(knowledge_data_training)
nrow(knowledge_data_testing)
We have now loaded the data. It was already separated into training and testing sets, however, some of the knowledge level categories are misnamed in the testing set, so we fixed this by renaming them. We have 258 training observations and 145 testing observations. We will now see if any of the rows contain any NA values.
map_df(knowledge_data_training, ~ any(is.na(.x)))
print('Table 3')
map_df(knowledge_data_testing, ~ any(is.na(.x)))
print('Table 4')
The training and the testing datasets contain no NA values. We are now going to see the mean values for each of the predictor columns:
knowledge_data_training_mean <- map_df(knowledge_data_training, ~ mean(., na.rm = TRUE)) %>% select(STG, SCG, STR, LPR, PEG)
knowledge_data_training_mean
print('Table 5')
We will also check the mode, or how common the values for UNS are:
knowledge_data_training_uns_count <- knowledge_data_training %>%
select(UNS) %>%
group_by(UNS) %>%
summarize(n = n()) %>%
arrange(desc(n))
knowledge_data_training_uns_count
print('Table 6')
The knowledge levels Middle and Low are most frequent, with 88 and 83 occurrences respectively. High is relatively frequent, with 63 occurrences. very_low is not as common, with only 24 occurrences. This may cause problems when creating our model, so to fix this, we will now upscale the data to have a balanced number of knowledge level observations.
upscale_recipe <- recipe(UNS ~ ., data = knowledge_data_training) %>%
step_upsample(UNS, over_ratio = 1, skip = FALSE) %>%
prep()
knowledge_data_training_upsample <- bake(upscale_recipe, knowledge_data_training)
knowledge_data_training_upsample_count <- knowledge_data_training_upsample %>%
select(UNS) %>%
group_by(UNS) %>%
summarize(n = n()) %>%
arrange(desc(n))
knowledge_data_training_upsample_count
print('Table 7')
We are now going to see the correlations between these variables to pick out a few relevant ones. We can not use ggpairs() because the knowledge level is a categorical variable. Instead, we are going to create bar plots for each of the predictor variables against the knowledge level:
plot_STG <- knowledge_data_training %>%
group_by(UNS) %>%
summarize(avg = mean(STG)) %>%
mutate(UNS = fct_relevel(UNS, "very_low", "Low", "Middle", "High")) %>%
ggplot(aes(x = UNS, y = avg, fill = UNS)) +
geom_bar(stat = "identity", position = "stack") +
labs(x = "Knowledge level", y = "Degree of study time for goal object materials", fill = "Knowledge level", title = "Average study time for goal object materials and knowledge level")
plot_STG
print('Fig. 1')
In this graph we examine the mean of the degree of study time for goal object materials for each of the knowledge levels. We can see as the knowledge level increases, the average degree of study time for goal object materials increases as well. This indicates that this will be a good predictor variable in our model.
plot_SCG <- knowledge_data_training %>%
group_by(UNS) %>%
summarize(avg = mean(SCG)) %>%
mutate(UNS = fct_relevel(UNS, "very_low", "Low", "Middle", "High")) %>%
ggplot(aes(x = UNS, y = avg, fill = UNS)) +
geom_bar(stat = "identity", position = "stack") +
labs(x = "Knowledge level", y = "Degree of repetition of goal object materials", fill = "Knowledge level", title = "Average repetition of goal object materials and knowledge level")
plot_SCG
print('Fig. 2')
In this graph we examine the mean of the degree of repetition of goal object materials for each of the knowledge levels. We can see as the knowledge level increases, the average degree of repetition of goal object materials increases as well. This indicates that this will also be a good predictor variable in our model.
plot_STR <- knowledge_data_training %>%
group_by(UNS) %>%
summarize(avg = mean(STR)) %>%
mutate(UNS = fct_relevel(UNS, "very_low", "Low", "Middle", "High")) %>%
ggplot(aes(x = UNS, y = avg, fill = UNS)) +
geom_bar(stat = "identity", position = "stack") +
labs(x = "Knowledge level", y = "Degree of study time for related objects with goal object", fill = "Knowledge level", title = "Average study time for related objects with goal object and knowledge level")
plot_STR
print('Fig. 3')
Unlike the first two variables, the mean of the degree of study time for related objects with goal object does not always increase with the different knowledge levels. Therefore, this variable doesn't seem appropriate for our model.
plot_LPR <- knowledge_data_training %>%
group_by(UNS) %>%
summarize(avg = mean(LPR)) %>%
mutate(UNS = fct_relevel(UNS, "very_low", "Low", "Middle", "High")) %>%
ggplot(aes(x = UNS, y = avg, fill = UNS)) +
geom_bar(stat = "identity", position = "stack") +
labs(x = "Knowledge level", y = "Exam performance for related objects with goal object", fill = "Knowledge level", title = "Average exam performance for related objects with goal object and knowledge level")
plot_LPR
print('Fig. 4')
This graph indicates there the mean of exam performance for related objects with goal object does not increase when knowledge level increases. Therefore, this variable doesn't seem appropriate for our model.
plot_PEG <- knowledge_data_training %>%
group_by(UNS) %>%
summarize(avg = mean(PEG)) %>%
mutate(UNS = fct_relevel(UNS, "very_low", "Low", "Middle", "High")) %>%
ggplot(aes(x = UNS, y = avg, fill = UNS)) +
geom_bar(stat = "identity", position = "stack") +
labs(x = "Knowledge level", y = "Exam performance for goal objects", fill = "Knowledge level", title = "Average exam performance for goal objects and knowledge level")
plot_PEG
print('Fig. 5')
This graph indicates that exam performance for goal objects greatly increases when knowledge level increases. This is a very good predictor variable for knowledge level. Because of this, we will use this in our final model.
The preliminary exploratory data analysis indicates that we should use three of the five potential predictor variables in our final model: Degree of study time for goal object materials (STG), Degree of repetition of goal object materials (SCG), and Exam performance for goal objects (PEG). The three predictor variables will be used in our final model to classify Knowledge levels (UNS).
Methods: Classification Model¶
Now that we have tidied (steady nomenclature and no NA values) and modified (upscaled) our data to our model's needs and decided our predictor variables, we'll move on to the model. We will be using k-nearest neighbors classification to create our tuning model, and will then make a final predictive one.
We'll start by selecting our predictor variables for convenience.
#Choosing the cols we are going to use
knowledge_train<-knowledge_data_training_upsample%>%
select(UNS,STG,SCG,PEG)
knowledge_data_test<-knowledge_data_testing%>%
select(UNS,STG,SCG,PEG)
We will then follow the standard procedure for k-nn classification - create a model specification and a recipe, and then input it into a workflow. However, we wish to know which value of the parameter k (the number of classifiers) would be most accurate. So, we tune the classifier using 10-fold cross-validation, and that will eventually lead us to the accuracy of different values of k. We also standardize all the variables to make sure they are comparable.
gridvals = tibble(neighbors = seq(from = 1, to = 10, by = 1))
knowledge_vfold <- vfold_cv(knowledge_train, v = 10, strata = UNS)
knn_spec <- nearest_neighbor(weight_func = "rectangular", neighbors = tune()) %>%
set_engine("kknn") %>%
set_mode("classification")
knowledge_recipe <- recipe(UNS ~ STG + SCG + PEG, data = knowledge_train) %>%
step_scale(all_predictors()) %>%
step_center(all_predictors())
knn_results <- workflow() %>%
add_recipe(knowledge_recipe) %>%
add_model(knn_spec) %>%
tune_grid(resamples = knowledge_vfold, grid = gridvals) %>%
collect_metrics() %>%
filter(.metric == "accuracy") %>%
arrange(desc(mean))
head(knn_results)
print('Table 8')
We will now plot the mean accuracy of the k's.
accuracies <- knn_results %>%
filter(.metric == 'accuracy')
options(repr.plot.height = 6, repr.plot.width = 10)
accuracy_vs_k <- ggplot(accuracies, aes(x = neighbors, y = mean)) +
geom_point() +
geom_line() +
labs(x = 'Neighbors', y = 'Accuracy Estimate')
accuracy_vs_k
print('Fig. 6')
As seen in the tibble as well as the graph, both k = 1 and k = 2 have the highest and exact same accuracy. So, to distinguish which would be better, we will plot graphs for k = 1 and k = 2. From there, we will determin the best k value for our final model.
knn_spec1 <- nearest_neighbor(weight_func = "rectangular", neighbors = 1) %>%
set_engine("kknn") %>%
set_mode("classification")
stg_grid <- seq(min(knowledge_train$STG), max(knowledge_train$STG), length.out = 100)
scg_grid <- seq(min(knowledge_train$SCG), max(knowledge_train$SCG), length.out = 100)
tcgrid1 <- as_tibble(expand.grid(STG=stg_grid, SCG=scg_grid))
#k=1 STG and SCG
knowledge_recipe1 <- recipe(UNS ~ STG + SCG, data = knowledge_train) %>%
step_scale(all_predictors()) %>%
step_center(all_predictors())
knn_fit1 <- workflow() %>%
add_recipe(knowledge_recipe1) %>%
add_model(knn_spec1) %>%
fit(data = knowledge_train)
prediction_table1 <- predict(knn_fit1, tcgrid1) %>% bind_cols(tcgrid1)
#k=2 STG and SCG
knn_spec2 <- nearest_neighbor(weight_func = "rectangular", neighbors = 2) %>%
set_engine("kknn") %>%
set_mode("classification")
knn_fit2 <- workflow() %>%
add_recipe(knowledge_recipe1) %>%
add_model(knn_spec2) %>%
fit(data = knowledge_train)
prediction_table2 <- predict(knn_fit2, tcgrid1) %>% bind_cols(tcgrid1)
options(repr.plot.height = 6, repr.plot.width = 10)
cbPalette <- c("#000000", "#E69F00", "#D55E00", "#009E73", "#CC79A7", "#F0E442", "#0072B2", "#56B4E9")
wkflw_plot1 <- ggplot() +
geom_point(data = knowledge_train, mapping = aes(x = STG, y = SCG, color = UNS), alpha=0.75) +
geom_point(data = prediction_table1, mapping = aes(x = STG, y = SCG, color = .pred_class), alpha=0.03, size=5.)+
labs(color = "UNS", x= "Degree of study time for goal object materials", y= "Degree of repetition of goal object materials") +
ggtitle("K=1, Degree of study time for goal object materials and\nDegree of repetition of goal object materials\npredicting Knowledge level") +
scale_color_manual(labels = c("Very Low", "Low", "Middle", "High"), values = cbPalette) +
theme(text = element_text(size = 20))
wkflw_plot2 <- ggplot() +
geom_point(data = knowledge_train, mapping = aes(x = STG, y = SCG, color = UNS), alpha=0.75) +
geom_point(data = prediction_table2, mapping = aes(x = STG, y = SCG, color = .pred_class), alpha=0.03, size=5.)+
labs(color = "UNS", x= "Degree of study time for goal object materials", y= "Degree of repetition of goal object materials") +
ggtitle("K=2, Degree of study time for goal object materials and\nDegree of repetition of goal object materials\npredicting Knowledge level") +
scale_color_manual(labels = c("Very Low", "Low", "Middle", "High"), values = cbPalette) +
theme(text = element_text(size = 20))
Below are the plots of Degree of repetition of goal object materials vs. the Degree of study time for goal object materials for k = 1 and k = 2.
wkflw_plot1
print('Fig. 7')
wkflw_plot2
print('Fig. 8')
knn_spec3 <- nearest_neighbor(weight_func = "rectangular", neighbors = 1) %>%
set_engine("kknn") %>%
set_mode("classification")
stg_grid3 <- seq(min(knowledge_train$STG), max(knowledge_train$STG), length.out = 100)
peg_grid3 <- seq(min(knowledge_train$PEG), max(knowledge_train$PEG), length.out = 100)
spgrid3 <- as_tibble(expand.grid(STG=stg_grid3, PEG=peg_grid3))
#k=1 STG and PEG
knowledge_recipe3 <- recipe(UNS ~ STG + PEG, data = knowledge_train) %>%
step_scale(all_predictors()) %>%
step_center(all_predictors())
knn_fit3 <- workflow() %>%
add_recipe(knowledge_recipe3) %>%
add_model(knn_spec3) %>%
fit(data = knowledge_train)
prediction_table3 <- predict(knn_fit3, spgrid3) %>% bind_cols(spgrid3)
#k=2 STG and PEG
knowledge_recipe4 <- recipe(UNS ~ STG + PEG, data = knowledge_train) %>%
step_scale(all_predictors()) %>%
step_center(all_predictors())
knn_fit4 <- workflow() %>%
add_recipe(knowledge_recipe3) %>%
add_model(knn_spec3) %>%
fit(data = knowledge_train)
prediction_table4 <- predict(knn_fit4, spgrid3) %>% bind_cols(spgrid3)
options(repr.plot.height = 6, repr.plot.width = 10)
wkflw_plot3 <- ggplot() +
geom_point(data = knowledge_train, mapping = aes(x = STG, y = PEG, color = UNS), alpha=0.75) +
geom_point(data = prediction_table3, mapping = aes(x = STG, y = PEG, color = .pred_class), alpha=0.03, size=5.)+
labs(color = "UNS", x = "Degree of study time for goal object materials", y = "Exam performance for goal objects") +
ggtitle("K=1, Degree of study time for goal object materials and\nExam performance for goal objects\npredicting Knowledge level") +
scale_color_manual(labels = c("Very Low", "Low", "Middle", "High"), values = cbPalette) +
theme(text = element_text(size = 20))
options(repr.plot.height = 6, repr.plot.width = 10)
wkflw_plot4 <- ggplot() +
geom_point(data = knowledge_train, mapping = aes(x = STG, y = PEG, color = UNS), alpha=0.75) +
geom_point(data = prediction_table4, mapping = aes(x = STG, y = PEG, color = .pred_class), alpha=0.03, size=5.)+
labs(color = "UNS", x = "Degree of study time for goal object materials", y = "Exam performance for goal objects") +
ggtitle("K=2, Degree of study time for goal object materials and\nExam performance for goal objects\npredicting Knowledge level") +
scale_color_manual(labels = c("Very Low", "Low", "Middle", "High"), values = cbPalette) +
theme(text = element_text(size = 20))
Below are the plots of Exam performance of goal object materials vs. the Degree of study time for goal object materials.
wkflw_plot3
print('Fig. 9')
wkflw_plot4
print('Fig. 10')
From these plots, we can observe that k = 1 and k = 2 produce almost the same models. There seems to be a very small insignificant change in the first two graphs at the bottom left coroner. This change is so small and so insignificant that is does not matter what k we choose. In the future, models may need to account for the differences between the two choices, and as a result, build two models for k = 1 and k = 2. We decided to make k = 2.
Results: Testing¶
We will now make our final model and attempt to predict on our testing data.
#work flow
knn_spec <- nearest_neighbor(weight_func = "rectangular", neighbors = 2) %>%
set_engine("kknn") %>%
set_mode("classification")
knowledge_recipe <- recipe(UNS ~ STG + SCG + PEG, data = knowledge_train) %>%
step_scale(all_predictors()) %>%
step_center(all_predictors())
knn_fit_final <- workflow() %>%
add_recipe(knowledge_recipe) %>%
add_model(knn_spec) %>%
fit(data = knowledge_train)
predictions <- predict(knn_fit_final, knowledge_data_test) %>%
bind_cols(knowledge_data_test)
training_results_metrics <- predictions %>%
metrics(truth = UNS, estimate = .pred_class)
training_results_conf_mat <- predictions %>%
conf_mat(truth = UNS, estimate = .pred_class)
training_results_metrics
print('Table 9')
training_results_conf_mat
We attained an accuracy of close to 72% in our final model on the test data.
We will now visualize the accuracy of the model. Because we have three predictor variables, we will need to create two separate models, and two separate plots to visualize them. On the plots, The color of the lightly shaded area represents the predicted knowledge level from the model, and the color of the points are the test points levels.
We will plot first plot Degree of study time for goal object materials vs Degree of repetition of goal object materials, and compare the differences between the knowledge levels on the model and the test points.
#Test: k=2 STG and SCG
knn_spec <- nearest_neighbor(weight_func = "rectangular", neighbors = 2) %>%
set_engine("kknn") %>%
set_mode("classification")
knowledge_recipe <- recipe(UNS ~ STG + SCG, data = knowledge_train) %>%
step_scale(all_predictors()) %>%
step_center(all_predictors())
knn_fit_STG_vs_SCG <- workflow() %>%
add_recipe(knowledge_recipe) %>%
add_model(knn_spec) %>%
fit(data = knowledge_train)
stg_grid <- seq(min(knowledge_data_test$STG), max(knowledge_data_test$STG), length.out = 100)
scg_grid <- seq(min(knowledge_data_test$SCG), max(knowledge_data_test$SCG), length.out = 100)
tcgrid <- as_tibble(expand.grid(STG=stg_grid, SCG=scg_grid))
knnPredGrid <- predict(knn_fit_STG_vs_SCG, tcgrid)
prediction_table <- bind_cols(knnPredGrid, tcgrid)
options(repr.plot.height = 8, repr.plot.width = 10)
wkflw_plot <- ggplot() +
geom_point(data = knowledge_data_test, mapping = aes(x = STG, y = SCG, color = UNS), alpha=1) +
geom_point(data = prediction_table, mapping = aes(x = STG, y = SCG, color = .pred_class), alpha=0.03, size=5.)+
labs(color = "UNS", x= "Degree of study time for goal object materials", y= "Degree of repetition of goal object materials") +
ggtitle("Degree of study time for goal object materials and\nDegree of repetition of goal object materials and\nKnowledge level model accuracy") +
scale_color_manual(labels = c("Very Low", "Low", "Middle", "High"), values = cbPalette) +
theme(text = element_text(size = 20))
wkflw_plot
print('Fig. 11')
For the most part, many of the levels from the test set do align with the predicated levels. There is a large group of "Low" knowledge levels that are in the "High" prediction area. This may be because in this model did not account for the "Exam performance for goal objects", and as a result, did not accurately classify that group of points.
We will now plot first plot Degree of study time for goal object materials vs Exam performance for goal objects, and compare the differences between the knowledge levels on the model and the test points.
#Test: k=2 STG and SCG
knn_spec <- nearest_neighbor(weight_func = "rectangular", neighbors = 2) %>%
set_engine("kknn") %>%
set_mode("classification")
knowledge_recipe <- recipe(UNS ~ STG + PEG, data = knowledge_train) %>%
step_scale(all_predictors()) %>%
step_center(all_predictors())
knn_fit_STG_vs_PEG <- workflow() %>%
add_recipe(knowledge_recipe) %>%
add_model(knn_spec) %>%
fit(data = knowledge_train)
stg_grid <- seq(min(knowledge_data_test$STG), max(knowledge_data_test$STG), length.out = 100)
scg_grid <- seq(min(knowledge_data_test$PEG), max(knowledge_data_test$PEG), length.out = 100)
tcgrid <- as_tibble(expand.grid(STG=stg_grid, PEG=scg_grid))
knnPredGrid <- predict(knn_fit_STG_vs_PEG, tcgrid)
prediction_table_STG_vs_PEG <- bind_cols(knnPredGrid, tcgrid)
options(repr.plot.height = 8, repr.plot.width = 10)
wkflw_plot <- ggplot() +
geom_point(data = knowledge_data_test, mapping = aes(x = STG, y = PEG, color = UNS), alpha=1) +
geom_point(data = prediction_table_STG_vs_PEG, mapping = aes(x = STG, y = PEG, color = .pred_class), alpha=0.03, size=5.)+
labs(color = "UNS", x = "Degree of study time for goal object materials", y = "Exam performance for goal objects") +
ggtitle("Degree of study time for goal object materials and\nExam performance for goal objects\nKnowledge level model accuracy") +
scale_color_manual(labels = c("Very Low", "Low", "Middle", "High"), values = cbPalette) +
theme(text = element_text(size = 20))
wkflw_plot
print('Fig. 12')
We can see that some of the actual levels from the test points are not the same as the predicted levels from the model. However, for the most part, many of the levels from the test set do align with the predicated levels. Most of the differences on this plot are near the borders of the shaded area, which is to be expected for many classification models.
Discussion¶
We attained an accuracy of close to 72% for our model on the testing set. This means our model is correct at predicting the knowledge level of a user 72% of the times based on their study time, degree of repetition, and exam performance on a goal object's material. We hypothesized an accuracy somewhat like this, as the predictor variables chosen are conventionally regarded as appropriate ones to gauge how developed someone's knowledge on a topic is; but, due to individual differences it isn't always possible to accurately predict one's "knowledge levels", and also due to the fact that it's not an actual unit of measurement and is a uniquely made up score. Moreover, we recognize that the accuracy rate could potentially rise if the model could be built off of a larger data set - this was a relatively small dataset (only 403 entries), and perhaps one with over 1000-1500 entries could provide a more precise estimate. Therefore, 72% is somewhat seated in the mean and definitely not a bad accuracy rate when used on larger data as well.
The model is certainly extremely useful. It serves a purpose for both academics in pedagogy as well as educators. Teachers and professors can use this model to predict how well their students understand the course material with current teaching methods, and what strategies they could implement or change to potentially improve performance. This feedback could help the educator improve their course planning, and it would quite obviously be beneficial to the students. Pedagogical researchers could also use this model or a variation of this model when carrying out inferential analysis in their research, or cite it when attempting to test a theory.
While this model is effective, something else it brings to the table is questions for the future. Data scientists and academics in the future could look to investigating other how a few other - perhaps unconventional - variables could impact knowledge levels or retention. e.g. socioeconomic status, or some sort of emotionally-derived variable such as familial relations or marital status of parents. It goes without saying, however, that it is necessary to examine the correlations of these variables first. Another question this research poses is, what other ways can knowledge "levels" or retention be measured? Must it always be categorical? What are some variables other than "knowledge level" that could be indicative one's understanding or comprehension? It is hoped and expected that these questions will soon be answered or clarified.
References¶
Beers, G. W., & Bowden, S. (2005). The effect of teaching method on long-term knowledge retention. Journal of Nursing Education, 44(11), 511-514.
Kahraman, H. T. (2009). Designing and application of web-based adaptive intelligent education system (Doctoral dissertation, Ph. D. Thesis, Institute of Science and Technology, Ankara).
Mueller, P. A., & Oppenheimer, D. M. (2014). The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological science, 25(6), 1159-1168.